Real-Time Pose-Based Attention Analysis
Embedded CV • Pose estimation + interpretable attention heuristics on Raspberry Pi
Built an embedded system that captures live video, runs on-device pose estimation to extract 17 body keypoints, applies attention heuristics (head tilt, drooping, hand-raise), and flags “Attentive” vs “Not Attentive” in real time — all on a Raspberry Pi.
TL;DR
On-device MoveNet (TFLite) pose estimation + fast heuristics to classify attention status in real time on Raspberry Pi.
My role
Architected the full pipeline (capture → preprocessing → TFLite inference → keypoints → heuristics → overlay → CSV logging) and optimized performance via threaded capture.
Tech
Links
Features
- Real-time pose estimation (~15 FPS on Pi)
- Attention heuristics: head tilt, drooping, hand-raise detection
- On-device processing (no cloud required)
- Visual overlay with keypoints + status
- CSV logging for future learning-based upgrades
My role
- Built the end-to-end pipeline: video capture → preprocessing → TFLite inference → feature extraction → attention analysis → logging
- Integrated threaded capture (imutils) for smooth frame rates
- Implemented CSV logging of timestamp, head-tilt, attention status, feedback, and raw keypoints
- Optimized preprocessing to match the quantized MoveNet model
Challenges and solutions
- Pi performance: baseline ~5 FPS → improved using threaded capture and lighter preprocessing.
- Model dtype mismatch: adjusted preprocessing to satisfy quantized TFLite expectations.
- Heuristic tuning: iteratively tuned thresholds on real data to reduce false positives.
Outcomes
- Delivered an end-to-end demo running fully on Raspberry Pi
- Logged 10,000+ frames for future classifier training
- Benchmarked ~15 FPS real-time inference with 17 keypoints
Future enhancements
- Train a learned classifier using the logged dataset
- Add additional cues: body lean, eye gaze, facial expression
- Build a web dashboard for live monitoring + analytics
- Optional cloud streaming for centralized analytics
Repository
Explore the code on GitHub: Real-Time Pose-Based Attention Analysis